8 research outputs found
Rethinking the Discount Factor in Reinforcement Learning: A Decision Theoretic Approach
Reinforcement learning (RL) agents have traditionally been tasked with
maximizing the value function of a Markov decision process (MDP), either in
continuous settings, with fixed discount factor , or in episodic
settings, with . While this has proven effective for specific tasks
with well-defined objectives (e.g., games), it has never been established that
fixed discounting is suitable for general purpose use (e.g., as a model of
human preferences). This paper characterizes rationality in sequential decision
making using a set of seven axioms and arrives at a form of discounting that
generalizes traditional fixed discounting. In particular, our framework admits
a state-action dependent "discount" factor that is not constrained to be less
than 1, so long as there is eventual long run discounting. Although this
broadens the range of possible preference structures in continuous settings, we
show that there exists a unique "optimizing MDP" with fixed whose
optimal value function matches the true utility of the optimal policy, and we
quantify the difference between value and utility for suboptimal policies. Our
work can be seen as providing a normative justification for (a slight
generalization of) Martha White's RL task formalism (2017) and other recent
departures from the traditional RL, and is relevant to task specification in
RL, inverse RL and preference-based RL.Comment: 8 pages + 1 page supplement. In proceedings of AAAI 2019. Slides,
poster and bibtex available at
https://silviupitis.com/#rethinking-the-discount-factor-in-reinforcement-learning-a-decision-theoretic-approac
Fixed-Horizon Temporal Difference Methods for Stable Reinforcement Learning
We explore fixed-horizon temporal difference (TD) methods, reinforcement
learning algorithms for a new kind of value function that predicts the sum of
rewards over a number of future time steps. To learn the value
function for horizon , these algorithms bootstrap from the value function
for horizon , or some shorter horizon. Because no value function
bootstraps from itself, fixed-horizon methods are immune to the stability
problems that plague other off-policy TD methods using function approximation
(also known as "the deadly triad"). Although fixed-horizon methods require the
storage of additional value functions, this gives the agent additional
predictive power, while the added complexity can be substantially reduced via
parallel updates, shared weights, and -step bootstrapping. We show how to
use fixed-horizon value functions to solve reinforcement learning problems
competitively with methods such as Q-learning that learn conventional value
functions. We also prove convergence of fixed-horizon temporal difference
methods with linear and general function approximation. Taken together, our
results establish fixed-horizon TD methods as a viable new way of avoiding the
stability problems of the deadly triad.Comment: AAAI 202
Large Language Models Are Human-Level Prompt Engineers
By conditioning on natural language instructions, large language models
(LLMs) have displayed impressive capabilities as general-purpose computers.
However, task performance depends significantly on the quality of the prompt
used to steer the model, and most effective prompts have been handcrafted by
humans. Inspired by classical program synthesis and the human approach to
prompt engineering, we propose Automatic Prompt Engineer (APE) for automatic
instruction generation and selection. In our method, we treat the instruction
as the "program," optimized by searching over a pool of instruction candidates
proposed by an LLM in order to maximize a chosen score function. To evaluate
the quality of the selected instruction, we evaluate the zero-shot performance
of another LLM following the selected instruction. Experiments on 24 NLP tasks
show that our automatically generated instructions outperform the prior LLM
baseline by a large margin and achieve better or comparable performance to the
instructions generated by human annotators on 19/24 tasks. We conduct extensive
qualitative and quantitative analyses to explore the performance of APE. We
show that APE-engineered prompts can be applied to steer models toward
truthfulness and/or informativeness, as well as to improve few-shot learning
performance by simply prepending them to standard in-context learning prompts.
Please check out our webpage at
https://sites.google.com/view/automatic-prompt-engineer